Add SDMX metadata extractor step in sdmx pipeline#1842
Add SDMX metadata extractor step in sdmx pipeline#1842rohitkumarbhagat wants to merge 8 commits intodatacommonsorg:masterfrom
Conversation
rohitkumarbhagat
commented
Jan 15, 2026
rohitkumarbhagat
commented
- add run_from/run_until flags for running range of steps
Highlight key SDMX docs for faster navigation
Use extracted JSON for schema mapping and update tests
Document new pipeline step and outputs
Wire run_from/run_until into config and planning Document and test range filtering
Summary of ChangesHello @rohitkumarbhagat, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the SDMX import pipeline by introducing a dedicated step for metadata extraction and providing more flexible control over pipeline execution. The new metadata extraction step streamlines the processing of SDMX metadata by converting it to JSON, which improves efficiency for downstream operations like schema mapping. Additionally, the introduction of Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new metadata extraction step to the SDMX pipeline and adds run_from/run_until flags for more granular control over pipeline execution. The changes are well-implemented across the pipeline logic, tests, and documentation. I've identified a minor documentation issue and a regression in test coverage that should be addressed.
1 participant
